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Amendments to the Claims : 

This listing of claims replaces all prior versions and listings 
of claims in the application: 

Listing of Claims : 

1. (cancelled) 

2 . (previously presented) The method of claim 12 wherein 
generating the three dimensional model comprises identifying a 
structure in the range sensor information, identifying different 
sections of the structure, selecting geometric primitives for 
the different sections of the structure based at least in part 
on input from a person regarding different shapes of the 
different sections, and parametric fitting of the geometric 
primitives to the range sensor information. 

3. (previously presented) The method of claim 2, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information . 
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4. (original) The method of claim 2, wherein the at least 
one image sensor comprises multiple image sensors, and 
generating the three dimensional model further comprises 
refining the three dimensional model based on object surfaces 
mapped from images acquired by the image sensors. 

5. (previously presented) The method of claim 12, wherein 
generating the three dimensional model further comprises 
projecting and resampling points in the range sensor information 
onto a regular grid at a user-defined resolution to produce the 
height field. 

6. (original) The method of claim 5, wherein generating 
the three dimensional model further comprises processing the 
height field using hole-filling and tessellation to generate a 
triangle mesh representation of the three dimensional model. 

7. (original) The method of claim 2, wherein the range 
sensor information comprises light detection and ranging (LIDAR) 
sensor data from an active airborne laser sensor. 

8. (previously presented) The method of claim 12, wherein 
tracking orientation information comprises tracking position and 
orientation information of the at least one image sensor by 
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estimating a camera pose based at least in part on three 
dimensional parameters of point and line structures visible in 
the camera image, and projecting real-time video imagery 
information comprises projecting real-time video imagery 
information from the at least one image sensor onto the three 
dimensional model based on the tracked orientation information. 

9. (original) The method of claim 8, wherein tracking 
position and orientation information of the at least one image 
sensor further comprises processing data from a tracking sensor 
system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation sensor 
input . 

10. (previously presented) The method of claim 12, wherein 
the at least one image sensor comprises multiple image sensors, 
and projecting the real-time video imagery information comprises 
projecting multiple video streams from the multiple image 
sensors onto the three dimensional model . 

11. (cancelled) 

12. (previously presented) A method comprising: 
generating a three dimensional model of an environment from 
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range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein projecting the real-time video imagery information 
comprises generating a depth map image from a video sensor 
viewpoint, and projective texture mapping the real-time video 
imagery information onto the three dimensional model conditioned 
upon visibility as determined from the generated depth map 
image ; and 

wherein generating the depth map image and projective 
texture mapping the real-time video imagery information are 
performed using a one-pass approach on graphics hardware that 
supports SGI OpenGL extensions. 

13. (previously presented) The method of claim 12, wherein 
visualizing the three dimensional model comprises: 
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video-projecting onto a display screen using a stereo 
video-projector; and 

coupling a rendering viewpoint to a user's head position 
using data from a tracker. 

14. (cancelled) 

15. (previously presented) The system of claim 22, wherein 
the at least one image sensor comprises multiple image sensors. 

16. (currently amended) The system of claim 15 [[16]], 
wherein the real-time video imagery information comprises pre- 
recorded real-time video imagery information. 

17. (original) The system of claim 15, wherein the dynamic 
fusion imagery projection component bases the real-time video 
imagery projection on a viewpoint separate from viewpoints 
associated with the multiple image sensors. 

18. (previously presented) The system of claim 22 wherein 
the model construction component performs operations comprising: 

identifying a structure in the range sensor information; 
identifying different sections of the structure; 
selecting geometric primitives for the different sections 
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of the structure based at least in part on input from a person 
regarding different shapes of the different sections; and 

parametric fitting of the selected geometric primitives to 
the range sensor information. 

19. (original) The system of claim 18, wherein the 
operations further comprise: 

projecting and resampling points in the range sensor 
information onto a regular grid at a user-defined resolution to 
produce the height field; and 

processing the height field using hole-filling and 
tessellation to generate a triangle mesh representation of the 
three dimensional model. 

20. (previously presented) The system of claim 22, wherein 
the at least one image sensor comprises multiple image sensors, 
and the system further comprises a model refinement component 
that refines the three dimensional model based on object 
surfaces mapped from images acquired by the image sensors. 

21. (previously presented) The system of claim 22, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain position and 
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orientation information of the at least one image sensor, and 
the dynamic fusion imagery projection component projects the 
real-time video imagery information based on the position and 
orientation information. 

22 . (previously presented) An augmented virtual 
environment system comprising: 

a model construction component that generates a three 
dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the visualization sub-system comprises the dynamic 
fusion imagery projection component and graphics hardware that 
supports SGI OpenGL extensions, and uses a one-pass approach on 
the graphics hardware to generate a depth map image from a video 
sensor viewpoint and projective texture map the real-time video 
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imagery information to project the real-time video imagery- 
information conditioned upon visibility as determined from the 
generated depth map image. 

23. (previously presented) The system of claim 22, wherein 
the visualization sub-system comprises: 

a stereo video-projector; and 
a tracker. 

24. (cancelled) 

25. (previously presented) The method of claim 29, wherein 
the surface comprises a two dimensional surface. 

26. (original) The method of claim 25, wherein placing the 
two dimensional surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 
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27. (cancelled) 

28. (cancelled) 

29. (previously presented) A method comprising: 
obtaining a three dimensional model of an environment; 
identifying in real time a region in motion with respect to 

a background image in real-time video imagery information from 
at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model, the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery information; 

placing a surface that corresponds to the moving region in 
the three dimensional model; 

projecting the real-time video imagery information onto the 
three dimensional model, including the surface, based on the 
position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein identifying a region in motion in real time 
comprises subtracting the background image from the real-time 
video imagery information, identifying a foreground object in 
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the subtracted real-time video imagery information, validating 
the foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent 
image frames in the real-time video imagery information. 

30. (previously presented) The method of claim 29, further 
comprising tracking the position and orientation information of 
the at least one image sensor in the environment with respect to 
the three dimensional model in real-time. 

31. (original) The method of claim 30, wherein obtaining a 
three dimensional model of an environment comprises generating 
the three dimensional model of the environment from range sensor 
information representing a height field for the environment. 

32. (cancelled) 



Page: 11 of 24 



Applicant : Ulrich Neumann et al . 
Serial No. : 10/676, 377 
Filed: September 30, 2003 



Attorney's Docket No.: 06666-156001 
USC-3345 



33. (previously presented) The system of claim 37, wherein 
the surface comprises a two dimensional surface. 

34. (original) The system of claim 33, wherein the object 
detection and tracking component places the two dimensional 
surface by performing operations comprising: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

35. (cancelled) 

36. (cancelled) 

37. (previously presented) An augmented virtual 
environment system comprising: 

an object detection and tracking component that identifies 
in real time a region in motion with respect to a background 
image in real-time video imagery information from at least one 
image sensor having associated position and orientation 
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information with respect to a three dimensional model, the 
background image comprising a single distribution background 
dynamically modeled from a time average of the real-time video 
imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model; 

a dynamic fusion imagery projection component that projects 
the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position 
and orientation information; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the object detection and tracking component 
identifies the moving region by performing operations comprising 
subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the 
foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; and 
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wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent 
image frames in the real-time video imagery information. 

38. (previously presented) The system of claim 37, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain the position and the 
orientation information associated with the at least one image 
sensor in real time in conjunction with the real-time video 
imagery. 

39. (original) The system of claim 38, further comprising 
a model construction component that generates the three 
dimensional model of an environment from range sensor 
information representing a height field for the environment. 

40. (cancelled) 

41. (cancelled) 

42. (cancelled) 

43. (cancelled) 
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44. (cancelled) 

45. (previously presented) The method of claim 3, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

46. (previously presented) The system of claim 18, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information . 

47. (previously presented) The system of claim 46, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 



Page: 15 of 24 



Applicant : Ulrich Neumann et al . 
Serial No. : 10/676, 377 
Filed: September 30, 2003 



Attorney's Docket No.: 06666-156001 
USC-3345 



a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

48. (cancelled) 

49. (cancelled) 
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